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1.  Introduction 


To  increase  readiness  and  capabilities,  the  U.S.  Army  is  constantly  introducing  newly  evolving 
weapons  and  technology  systems.  However,  in  doing  so,  actual  system  performance  does  not 
always  meet  expectations.  Additionally,  the  replacement  or  product  improvement  of  existing 
systems  often  requires  additional  training,  skills,  or  Soldiers  to  operate,  maintain,  and  support  the 
systems.  This,  in  turn,  would  require  recruiting  more  highly  skilled  Soldiers,  additional  training 
and  training  expenses,  and  an  overall  expansion  of  Army  training  programs.  Moreover, 
increases  in  system  complexity  impact  mobilization,  readiness,  and  sustainability.  Design  details 
and  human  performance  of  emerging  systems  have  a  significant  impact  on  these  factors. 

One  of  the  U.S.  Army  Research  Laboratory’s  (ARL’s)  Major  Laboratory  Programs  (MLP), 
Human  Dimension  (HD),  was  conceived  to  address  these  issues.  This  MLP,  which  was 
formalized  by  the  U.S.  Army  Training  and  Doctrine  Command,  is  a  major  Army  effort  designed 
to  support  evaluations  of  the  integration  between  Soldiers  and  communications  systems, 
weapons,  and  vehicles  (U.S.  Army  Research  Laboratory,  2012). 

The  Human  Research  and  Engineering  Directorate  (HRED)  is  the  lead  organization  for  the  HD 
MLP.  Under  this  MLP,  HRED  is  responsible  for  evaluating  Soldier-system  performance  to 
make  certain  Soldiers  are  equipped  with  systems  they  can  operate  proficiently  with  minimal  risk. 
This  responsibility  includes  the  development  of  MANPRINT  (Manpower  and  Personnel 
Integration)  evaluations  of  the  systems,  which  include  assessment  method  development, 
evaluations  of  Soldier-system  design,  and  Soldier  performance  assessments. 

To  ensure  that  these  issues  are  considered  within  system  development  during  each  life  cycle 
phase,  these  evaluations  are  performed  during  the  System  Acquisition  Process.  These 
evaluations  are  used  to  influence  the  Milestone  Decision  Review  process  that  determines 
whether  or  not  the  system  is  ready  to  transition  to  the  next  scheduled  phase. 

MANPRINT  (The  MANPRINT  Mission,  2001)  is  a  comprehensive  management  domain 
designed  to  optimize  total  system  performance,  reduce  life  cycle  costs,  and  minimize  risk  of 
Soldier  loss  or  injury  by  focusing  on  human  requirements  and  considering  the  impact  of  materiel 
design  on  Soldiers  throughout  the  system  development  process.  A  major  thrust  of  MANPRINT 
is  to  identify  man-machine  interface  issues  which,  taken  individually  or  collectively,  may  be  so 
objectionable  that,  if  not  remedied,  would  warrant  a  decision  not  to  transition  into  the  next  phase. 

Identifying  and  rectifying  these  issues  is  achieved  by  thorough  investigation  of  the  following 
seven  MANPRINT  domains: 
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1.  Manpower  -  This  domain  assesses  the  number  of  military  and  civilian  personnel  required 
to  operate,  maintain,  sustain,  and  provide  training  for  systems  and  ensures  that  total 
manpower  requirements  lie  within  Army  constraints  and  resource  demands  of  the  system 
do  not  exceed  the  available  supply. 

2.  Personnel  -  This  domain  assesses  the  cognitive  and  physical  capabilities  required  to  be 
able  to  train  for,  operate,  maintain,  and  sustain  materiel  and  information  systems. 

3.  Training  -  This  domain  assesses  the  instruction  and  on-the-job  or  unit  training  required  to 
provide  personnel  with  essential  job  skills,  knowledge,  and  abilities.  It  ensures  that  the 
amount  and  content  of  training  will  allow  Soldiers  to  acquire  the  skills  necessary  to 
accomplish  their  tasks. 

4.  Human  Factors  Engineering  -  This  domain  assesses  the  integration  of  human 
characteristics  into  system  definition,  design,  development,  and  evaluation  to  optimize 
human-machine  performance  under  operational  conditions.  Considerations  of  this  domain 
include  making  the  equipment  easier  to  operate,  maintain,  and  support;  reducing  the  time 
required  to  complete  tasks;  reducing  operator  error;  and  reducing  time  and  money  spent  on 
training. 

5.  System  Safety  -  This  domain  assesses  the  design  features  and  operating  characteristics  of  a 
system  that  minimize  the  potential  for  human  or  machine  errors  or  failures  that  may  cause 
accidents  or  injuries  and  ensures  that  those  measures  are  designed  into  the  total  system. 

6.  Health  Hazards  -  This  domain  assesses  the  design  features  and  operating  characteristics 
of  a  system  that  create  significant  risks  of  bodily  injury  or  death  including  acoustics, 
biological  and  chemical  substances,  temperature  extremes  and  variations,  radiation,  oxygen 
deficiency,  shock  (not  electrical),  trauma,  and  vibration. 

7.  Soldier  Survivability  -  This  domain  assesses  the  characteristics  of  a  system  that  can 
reduce  fratricide,  detectability,  and  probability  of  being  attacked  and  minimize  system 
damage,  Soldier  injury,  and  cognitive  and  physical  fatigue. 

Throughout  the  design  and  development  phases,  MANPRINT  ensures  that  the  system  operation, 
maintenance,  training,  and  support  requirements  are  matched  to  personnel  availability  and 
capability;  systems  are  increasingly  user-friendly,  reliable,  and  maintainable;  and  system 
performance  is  optimized  at  minimal  life  cycle  costs. 
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2.  Methodology 


The  integration  team  of  the  MANPRINT  Methods  and  Analysis  Branch  (MMAB),  HRED,  is 
responsible  for  providing  MANPRINT  evaluation  support  to  the  U.S.  Department  of  Defense 
(DOD)  on  acquisition  programs.  These  evaluations  serve  to  optimize  system  performance  and 
minimize  life-cycle  costs,  ensuring  that  Soldier  and  organizational  needs  are  considered 
throughout  the  system  acquisition  process  and  are  integrated  into  the  system  design  while 
adhering  to  U.S.  Army  Regulation  (AR)  602-2  (2001). 

A  major  objective  of  this  effort  has  been  to  develop  and  execute  a  systems  engineering  approach 
along  with  standardized  field-operational  Soldier  performance  metrics  to  quantify  and  validate 
integrated  Soldier-information  systems  performance  on  the  digital  battlefield.  This  approach 
consists  of  three  steps:  (1)  identifying  tasks  and  behavioral  characteristics  associated  with 
effective  mission  command  performance  at  operator,  staff,  and  organizational  levels;  (2) 
developing  a  framework  for  measuring  the  usability,  functionality,  and  performance  of  the 
technology;  and  (3)  developing  an  evaluation  methodology  to  quantify  and  evaluate  Soldier- 
information  systems  performance,  which  is  the  focus  of  this  report  (Grynovicki  and  Branscome, 
to  be  published). 

2.1  Procedures 

MMAB  MANPRINT  assessments  strive  to  focus  on  the  Soldiers  by  talking  to  them,  determining 
what  they  want  and  need,  and  ensuring  they  are  adequately  trained  to  perform  their  mission. 

This  is  achieved  by  employing  multiple  evaluation  methods  to  ensure  that  the  Soldiers  are 
adequately  trained  to  perform  their  mission  and  that  the  equipment  is  easy  to  operate,  maintain, 
and  support,  while  possibly  reducing  the  time  to  accomplish  tasks,  the  chance  for  operator  error, 
the  amount  of  training  needed,  or  the  need  for  special  skills.  These  evaluation  methods  include 
observations,  questionnaires,  and  interviews  and  are  normally  performed  during  or  after  training 
and  during  or  after  task  execution. 

2.1.1  Observations 

Observation  methods  commonly  employed  include  heuristic  walk-throughs  and  over-the- 
shoulder  observations.  These  entail  subject  matter  experts  examining  training,  procedures,  and 
job  tasks  in  an  operational  setting  and  making  recommendations  based  on  key  usability 
principles  and  functionality  tasks.  When  space  considerations  allow,  HRED  practitioners  attend 
and/or  engage  in  actual  full  equipment  training,  including  classroom  and  hands-on  training.  This 
allows  first-hand  knowledge  of  the  system  and  opportunities  for  interacting  with  instructors  as 
well  as  users.  Evaluators  need  to  observe  the  training  in  the  field  and  talk  with  Soldiers  to  find 
out  how  well  the  system  met  their  needs  and  to  identify  any  issues  they  had  with  the  system. 
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Observations  in  operational  settings  allow  evaluators  to  see  first-hand  how  well  the  operators 
were  trained  on  the  system  and  determine  if  additional  training  would  be  required.  Additionally, 
it  affords  us  the  opportunity  to  examine  the  system’s  ability  to  provide  accurate  situational 
understanding  and  support  the  job  tasks  of  the  Soldier  so  they  can  complete  their  mission. 

2.1.2  Questionnaires 

Generally,  questionnaires  are  distributed  after  classroom  training,  hands-on  training,  and  during 
or  after  practical  exercises.  They  normally  consist  of  closed-ended  questions  and  Likert  scales 
based  on  key  factors  and  previously  defined  systems  requirements.  Question  items  can  be 
general  or  system-specific.  Questionnaire  responses  are  analyzed  to  obtain  the  Soldiers’  overall 
opinions,  recommendations,  and  concerns. 


An  example  of  general  questions  may  be  as  follows: 


Place  an  X  in  the  appropriate  box: 

Totally  Agree 

Agree 

Neither  Agree 
nor  Disagree 

Disagree 

Totally 

Disagree 

NA 

The  system  is  well  designed. 

— 

— 

— 

— 

— 

— 

The  system  is  reliable. 

— 

— 

— 

— 

— 

— 

The  system  is  easy  to  operate. 

— 

— 

— 

— 

— 

— 

Whereas,  an  example  of  system-specific  questions  may  be  as  follows: 


Place  an  X  in  the  appropriate  box: 

Totally  Agree 

Agree 

Neither  Agree 
nor  Disagree 

Disagree 

Totally 

Disagree 

NA 

It  was  easy  to  use  the  ABC  switch  to 

enable/disable  system  access. 

— 

— 

— 

— 

— 

— 

Plotting  IED  locations  was  easy, 
timely  and  accurate. 

— 

— 

— 

— 

— 

— 

A  complete  sample  questionnaire  is  included  in  appendix  A. 

2.1.3  Metrics 

Additionally,  a  select  battery  of  validated  metrics  is  used  to  measure  and  quantify  the  effects  of 
workload,  situation  awareness,  and  individual  differences  on  total  Soldier  performance  during 
the  MANPRINT  evaluations.  These  metrics  include  standardized  measures  with  demonstrated 
construct  validity  that  allow  us  to  collect  qualitative  and  quantitative  data  to  further  evaluate  the 
systems  in  terms  of  usability  and  to  support  the  overall  MANPRINT  assessment. 

2. 1.3.1  Workload.  Subjective  ratings  of  mental  and  physical  workload  provide  evaluators  with 
the  operators’  opinions  on  the  amount  of  effort  required  to  perform  tasks.  This  information  is 
vital  in  determining  the  amount  of  reduction  in  performance  that  may  be  attributed  to  increases 
in  workload. 
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The  National  Aeronautics  and  Space  Administration-Task  Load  Index  (NASA-TLX)  (Hart  and 
Staveland,  1988)  is  an  evaluation  of  the  relative  importance  of  six  factors  in  determining  how 
much  workload  an  individual  experiences  during  a  specific  task.  These  factors — mental  demand, 
physical  demand,  temporal  demand,  performance,  effort,  and  frustration  level — are  presented  as 
a  series  of  pairs.  The  participant  is  asked  to  choose  which  item  is  more  important  to  his  or  her 
experience  of  workload  during  the  task.  Choice  patterns  are  used  to  create  weighted 
combination  ratings  that  determine  summary  workload  scores.  Operators  are  then  asked  to  rate 
their  average  workload  experienced  during  the  mission  or  after  individual  tasks.  Resultant 
ratings  aid  researchers  in  evaluating  the  ease-of-use  and  suitability  of  the  interface.  The  NASA- 
TLX  has  been  used  in  evaluations  of  the  Force  XXI  Battle  Command  Brigade  and  Below 
(FBCB2)  (U.S.  Army  Evaluation  Center,  2011),  Future  Combat  Systems  (FCS)  Integrated 
Mission  Test  I  (IMT1)  (U.S.  Army  Test  and  Evaluation  Center,  2007),  and  Command,  Control, 
Communications,  Computers,  Intelligence,  Surveillance  and  Reconnaissance  systems.  The 
NASA-TLX  is  included  in  appendix  B. 

The  Modified  Cooper-Harper  (MCH)  scale  is  used  to  rate  operator  demand  level.  This  scale  is  a 
10-point  rating  scale  that  results  in  a  single  rating  of  workload  (Hill  et  al.,  1992).  This 
modification  of  the  Cooper-Harper  scale  allows  analysts  to  apply  it  to  Soldier-information 
systems  to  evaluate  how  well  the  operator  processes  display  information  and  identify 
deficiencies  in  operator  interfaces.  Most  commonly,  the  MCH  is  used  in  evaluating  operator 
interfaces  for  unmanned  vehicles,  such  as  the  Rabbit  2.0  system  (Scribner  et  al.,  to  be  published). 
A  variation  of  the  MCH  is  included  in  appendix  C. 

The  Situation  Awareness  Rating  Technique  (SART)  (Taylor,  1990)  provides  researchers  with  the 
operators’  subjective  opinions  on  their  workload  level  on  individual  tasks.  It  is  useful  in 
operational  environments  because  it  is  easy  to  use  and  administer  in  diversified  environments. 

Operators  rate  the  demand  on  their  resources,  supply  of  their  resources,  and  their  understanding 
of  the  situation  on  a  7-point  scale  from  “very  low”  to  “very  high.”  The  resulting  scores  are  then 
scaled  to  provide  the  evaluator  with  a  single  rating  of  overall  workload.  This  methodology  was 
also  used  successfully  in  the  FCS  IMT1.  The  Situation  Awareness  Rating  Technique  is  included 
in  appendix  D. 

2. 1.3. 2  Situation  Awareness.  Situation  Awareness  (SA)  is  a  three-level  concept:  (1)  knowing 
what  is  going  on  in  the  environment  (perception),  (2)  understanding  the  meaning  of  what  is 
going  on  (comprehension),  and  (3)  predicting  what  will  happen  in  the  future  based  on  the  current 
situation  (projection).  By  including  SA  analyses  during  the  MANPRINT  evaluation  process,  we 
are  able  to  increase  system  usability  and  enhance  human  factors  engineering  (HFE). 

SA  assessment  metrics  provide  us  with  objective  and  subjective  data  on  how  operators  and 
Soldiers  obtain,  comprehend,  and  act  on  information  and  allow  us  to  formulate  system  and 
informational  requirements.  The  Situation  Awareness  Global  Assessment  Technique  (SAGAT) 
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(Endsley,  2000)  is  used  to  measure  operator  SA  requirements  during  task  execution.  This  tool 
allows  evaluators  to  temporarily  stop  the  simulation  or  task  and  query  operators  about  their 
current  awareness  of  the  situation.  In  turn,  the  operators’  SA  can  then  be  compared  to  ground 
truth  to  provide  an  objective  measure  of  SA.  SAGAT  questions  are  customized  to  the  system 
being  evaluated.  Examples  of  SA  queries  may  be  found  in  appendix  E. 

Situation  Awareness  Behaviorally  Anchored  Rating  Scale  (Endsley,  1995)  is  an  objective 
measurement  tool  often  used  in  conjunction  with  SAGAT.  Evaluators  use  this  tool  to  observe 
and  rate  an  individual’s  behaviors  as  they  relate  to  SA  and  to  gain  insights  into  the  techniques 
and  strategies  used  by  the  operators.  The  tool  consists  of  28  items  that  have  been  identified  as 
being  significant  contributors  to  successful  SA  and  are  rated  on  a  5-point  scale. 

2.1.4  Interviews 

Interviews  are  structured  or  informal  discussion/question  and  answer  sessions  with  Soldiers,  key 
staff  leaders,  and  users.  When  feasible,  practitioners  engage  in  off-the-cuff  dialogue  with  users 
during  training  and  task  execution.  When  that  is  not  practical,  interviews  are  usually  afforded 
during  After  Action  Reviews  (AAR).  An  AAR  is  a  professional  discussion  between  developers, 
operators,  and  researchers  conducted  after  an  assessment  or  evaluation.  This  forum  allows 
researchers  the  opportunity  to  engage  in  discussions  with  operators,  maintainers,  and  developers 
to  determine  their  insights  and  specific  opinions,  recommendations,  and  concerns. 

Interviews  are  an  integral  component  of  the  overall  assessment  in  that  they  allow  the  researchers 
to  ask  what  happened  during  task  execution,  why  certain  events  happened,  and  how 
improvements  can  be  made  to  the  system  to  alleviate  problems  and  strengthen  weaknesses.  We 
use  these  opportunities  to  identify  tasks  or  procedures  that  require  additional  or  remedial 
training;  suggest  adjustments  or  improvements  to  the  training  materials;  make  recommendations 
for  system  improvements  or  modifications;  recommend  solutions  for  improving  HFE  and 
usability;  and  identify  lessons  learned  so  they  can  be  applied  to  future  applications. 


3.  Analysis  and  Recommendation 


Upon  analysis  of  evaluation  data,  issues,  ratings,  and  recommendations  for  each  MANPRINT 
domain  are  provided.  An  issue  can  impact  one  or  more  domains;  however,  it  is  normally  only 
addressed  under  the  domain  for  which  it  has  the  greatest  impact.  A  critical  issue  is  a  system 
characteristic  which,  if  not  remedied,  could  possibly  result  in  death  or  bodily  injury,  termination 
of  the  mission,  loss  of  the  system,  inability  of  the  system  to  perform  its  intended  mission,  or  an 
unacceptable  impact  on  the  manpower,  personnel,  or  training  requirements  of  the  system.  A 
major  issue  is  a  system  characteristic  which,  if  not  remedied,  could  possibly  result  in  bodily 
injury,  reduced  mission  performance,  extensive  system  damage,  seriously  diminished  capacity  of 
the  system  to  perform  its  intended  mission,  or  a  significant  negative  impact  on  the  manpower, 
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personnel,  or  training  requirements  of  the  system.  A  concern  is  a  system  characteristic  which,  if 
not  remedied,  could  possibly  result  in  discomfort  to  the  Soldier,  reduced  mission  effectiveness, 
or  system  damage.  The  color  ratings  by  domain  are  made  by  the  ARL-HRED  MANPRINT 
practitioner.  A  rating  of  green  indicates  that  the  system  is  ready  to  transition  to  the  next  level;  a 
rating  of  amber  indicates  that  the  system  has  minor  problems  that  should  be  addressed  but  are  not 
serious  enough  to  prevent  the  system  to  be  transitioned  to  the  next  level;  and  a  rating  of  red 
indicates  that  the  program  must  be  stopped  until  the  issue  is  resolved. 

Based  on  findings,  recommendations  for  changes  or  system  improvements  are  offered  by  the 
practitioner.  Recommendations  include,  but  are  not  limited  to,  the  following: 

1.  Assessing  and  determining  true  manpower  and  personnel  requirements  and  ensuring  that 
those  needs  lie  within  Army  constraints. 

2.  Providing  suggestions  for  enhanced  training  methodologies  and  improvements  to  training 
materials  and  ensuring  that  the  system  is  designed  for  the  target  population.  This  includes 
a  determination  of  how  much  and  what  type  of  training  is  necessary  to  provide  Soldiers 
with  the  skills  necessary  to  successfully  complete  their  tasks. 

3.  Determining  improved  operational  characteristics  of  the  system  under  consideration, 
including  controls,  displays,  the  man-machine  interface,  and  the  operational  and 
environmental  setting  in  which  the  system  is  operated. 

4.  Suggesting  improved  design  features  and  tactics,  techniques,  and  procedures  for  system 
employment  to  minimize  the  risk  of  potential  human  error,  system  malfunction,  mental  and 
physical  fatigue,  Soldier  detectability,  injury,  fratricide,  and  vulnerabilities  to  man  and 
machine. 


4.  Technology  Transition 


One  of  the  technical  objectives  of  the  HD  MLP  is  for  ARL  to  develop  tools  and  analytic 
methodologies  for  providing  MANPRINT  criteria  early  in  the  acquisition  lifecycle.  In  support  of 
that  objective,  MMAB  is  currently  involved  with  the  development  of  the  Metrics  of  Global 
Assessment  and  Situation  Awareness  (MEGA-SA)  (Bolstad,  2011)  data  collection  tool.  This 
tool  is  based  on  the  SAGAT  method  of  evaluation  and  will  allow  researchers  or  evaluators  to 
obtain  objective  measures  of  SA  in  real  time  at  all  levels  (perception,  comprehension,  projection) 
as  well  as  to  evaluate  training.  MEGA-SA  is  a  user-friendly  tool  that  will  be  used  in  laboratory 
and  field  evaluations.  It  will  provide  researchers  with  a  means  to  develop  customized  objective 
SA  assessments  across  Army  command  and  control  systems. 
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Evaluators  are  capable  of  customizing  assessments  to  meet  the  system’s  training  and 
performance  objectives  and  will  be  able  to  include  other  metrics  for  SA  and  workload  such  as 
NASA-TLX  as  well  as  customized  post-task  questionnaires. 

MEGA-SA  is  composed  of  four  modules:  Metric  Tool,  Setup  Tool,  Run  Tool,  and  Report  Tool. 
The  Metric  Tool  allows  users  to  create  a  database  of  metric  items,  response  items,  and  metric 
sets  to  be  used  during  a  specific  evaluation.  With  the  Setup  Tool,  evaluators  can  define  metric 
events  and  schedule  those  events  for  specific  or  random  times  during  the  evaluation.  It  allows 
the  user  to  determine  how  the  queries  will  be  presented  (visual  or  auditory),  who  the  queries  will 
be  presented  to,  how  often  the  queries  will  be  presented,  how  the  queries  are  triggered  (e.g.,  time 
stamp,  location,  or  event),  and  how  the  operator  is  expected  to  respond  to  the  queries.  The  Run 
Tool  allows  users  to  manage  and  monitor  the  events  and  monitor  the  network.  Lastly,  the  Report 
Tool  creates  reports,  analyzes  the  data,  and  exports  the  data  and  related  graphics. 

The  MEGA-SA  tool  will  be  an  invaluable  addition  to  the  MANPRINT  evaluation  process  as  it 
will  provide  MANPRINT  practitioners  with  a  flexible  customizable  tool  that  will  increase  the 
robustness  of  Army  systems  evaluations.  This  tool  will  be  adaptable  to  all  domains  throughout 
the  DOD. 


5.  Conclusion 


This  report  described  the  methodology  used  by  the  MANPRINT  MMAB,  ARL/HRED,  in 
performing  MANPRINT  Soldier- machine  evaluations  during  the  system  acquisition  process. 

This  global-based  approach  has  been  used  successfully  in  evaluations  of  many  Army  systems. 
ARL-HRED  supported  the  U.S.  Army  Test  and  Evaluation  Center  during  the  rapid  equipment 
fielding  of  the  Persistent  Surveillance  Dissemination  System  of  Systems  capability 
demonstration.  Quantitative  and  qualitative  data  analyses  using  the  methods  described  in  this 
report  lead  to  an  increased  capability  to  integrate  data-feed  from  imaging  sensors  as  well  as 
improvements  in  situational  awareness. 

Other  major  systems  evaluated  with  this  method  include  the  General  Fund  Enterprise  Business 
System,  Common  Remotely  Operated  Weapon  Station,  and  Net-Centric  Enterprise  Services, 
among  others.  By  combining  observation,  interview,  and  questionnaire  evaluation  methods, 
including  standardized  metrics  for  assessing  SA  and  workload,  we  are  able  to  provide  the  project 
manager  with  recommendations  and  establish  system  requirements  throughout  the  design  and 
development  phases,  thus  alleviating  extraneous  cost  and  time. 
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Appendix  A.  Sample  Manpower  and  Personnel  Integration  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Instructions 


The  purpose  of  this  questionnaire  is  to  collect  training  information  on  the  system.  Your  answers 
will  be  treated  confidentially  and  will  not  be  shown  to  anyone  except  those  who  are  evaluating 
the  system  for  the  Army.  None  of  your  information  will  be  given  to  your  chain  of  command  or 
put  in  your  personnel  file.  Please  fill  out  the  questionnaire  carefully.  If  you  need  additional 
space  to  answer  a  question,  indicate  by  an  arrow  (— >)  and  continue  on  the  back  of  the  page. 
Please  be  sure  to  number  the  item  on  the  back  of  the  page.  If  you  have  any  questions  please 
contact  a  test  team  representative  for  help.  Thank  you  for  your  assistance 

1)  Date: _ .  / _ .  / 

(DD/MMM/YY  Example:  01  Sep  11) 

3)  Grade _ 

4)  Primary  MOS _ 

5)  Time  in  Primary  MOS _ 

6)  Current  duty  position  _ _ 

7)  Time  in  current  duty  position  _ _ 


MANPOWER  &  PERSONNEL 

Answer  the  following  Manpower  &  Personnel  questions: 


(place  an  X  in  the  appropriate  box) 

Yes 

No 

a)  Is  your  skill  level  adequate  to  effectively  operate  the  system?  (If  no,  please  comment.) 

b)  Is  your  skill  level  adequate  to  effectively  maintain  the  system?  (If  no,  please  comment.) 

c)  Is  your  skill  level  adequate  to  effectively  perform  the  mission?  (If  no,  please  comment.) 

d)  Do  the  personnel  skills  required  to  set  up  and  operate  the  system  necessitate  a  specific 
Additional  Skills  Identifier  (ASI)  or  MOS?  (If  yes,  please  comment) 

e)  Did  you  require  additional  personnel  to  set  up  and  operate  the  system?  (If  yes,  please 
comment) 

2)  PIN: _ 

(Assigned  by  test  team) 


Comments: 
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The  number  of  operators  allocated  to  perform  the  installation/operation  tasks  is/are  adequate. 


Completely 

Disagree 

Moderately 

Disagree 

Slightly 

Disagree 

Slightly 

Agree 

Moderately 

Agree 

Completely 

Agree 

If  you  disagree  at  all,  please  explain. 


Soldiers  operating  this  system  will  need  a  specific  MOS  or  Additional  Skill  Identifier  (ASI). 


Completely 

Disagree 

Moderately 

Disagree 

Slightly 

Disagree 

Slightly 

Agree 

Moderately 

Agree 

Completely 

Agree 

Operate 

Maintain 

TRAINING 

Rate  the  operator  classroom  training  using  the  scale  below: 


(place  an  X  in  the  appropriate  box) 

Very 

Poor 

Poor 

Acceptable 

Good 

Very 

Good 

NA 

Length  of  training. 

Pace  of  training. 

Clarity  of  text. 

Completeness. 

Accuracy. 

Logical  sequence. 

Ease  of  reading. 

Ease  of  use. 

The  user  manuals  and  training  handouts  were 
easy  to  understand  and  organized  in  a  manner 
that  enabled  me  to  find  information  quickly. 

The  technical  and/or  user  manuals  adequately 
describe  how  to  install  and  operate  the  system. 

Comments: 
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Rate  the  hands-on  training  using  the  scale  below  and  provide  comments. 


(If  you  did  not  receive  training,  check  here _ and  go  to  the  next  question) 


(place  an  X  in  the  appropriate  box) 

Very 

Poor 

Poor 

Acceptable 

Good 

Very 

Good 

NA 

Completeness 

Accuracy 

Logical  sequence 

Installation  of  system 

Operation  and  control  of  system 

Practical  exercises 

The  pace  at  which  the  new  material  presented. 

Upon  completion  of  training,  I  feel  confident  in  my 
ability  to  complete  my  basic  mission. 

Overall  length  of  training. 

Overall  quality  of  training 

Training  Comments: 


Did  the  classroom/hands-on  training  address  safety  issues  associated  with  the  AHD? 


If  no,  please  comment. 


The  training  enabled  me  to  effectively  setup,  restore/repair,  operate,  troubleshoot,  maintain, 
shutdown  or  request  assistance  when  required. 


Completely 

Disagree 

Moderately 

Disagree 

Slightly 

Disagree 

Slightly 

Agree 

Moderately 

Agree 

Completely 

Agree 

Setup/  Startup 

Restore/ 

Repair 

Operate 

Troubleshoot 

Maintain 

Request 

Assistance 

14 


If  you  disagree  with  any  topic(s),  please  explain. 


The  technical  manual  (TM)  contained  just  the  right  amount  of  information  to  enable  me  to 
complete  my  tasks. 


Completely 

Disagree 

Moderately 

Disagree 

Slightly 

Disagree 

Slightly 

Agree 

Moderately 

Agree 

Completely 

Agree 

If  you  disagree,  please  explain. 


The  TM  was  accurate  and  contained  the  proper  task  execution  order  that  enabled  me  to 
troubleshoot,  diagnose  and  complete  my  tasks  in  an  organized  and  thorough  manner. 


Completely 

Disagree 

Moderately 

Disagree 

Slightly 

Disagree 

Slightly 

Agree 

Moderately 

Agree 

Completely 

Agree 

Troubleshoot 

Fault  Isolate 

Repair/Replace 

If  you  disagree  at  all,  please  explain. 


What  specific  areas  of  training  could  be  improved,  and  how  (i.e.  more  detailed  explanation, 
etc.)? 
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HFE  (Human  Factors  Engineering) 

Rate  the  following  HFE  characteristics. 


(place  an  X  in  the  appropriate  box) 

Very 

Poor 

Poor 

Acceptable 

Good 

Very 

Good 

NA 

Ability  to  start  the  system. 

Ability  to  use  the  x  control. 

Ability  to  use  the  y  control. 

System  response  time 

Size  of  displays  appropriate  for  intended  usage 

Visibility  of  all  displays  during  day  operations. 

Visibility  of  all  displays  during  night  operations. 

Shape  of  displays  appropriate  for  intended  usage. 

Illumination  of  displays  (screen 
brightness/contrast). 

Viewing  angle  of  displays. 

Use  of  displays  when  wearing  corrective  lenses. 

While  using  the  system  did  you  experience  any  of  the  following? 


(place  an  X  in  the  appropriate  box) 

Yes 

No 

NA 

a) 

Eye  strain?  (If  yes,  please  comment.) 

b) 

Tunnel  vision?  (If  yes,  please  comment.) 

c) 

Motion  sickness?  (If  yes,  please  comment.) 

d) 

Headaches?  (If  yes,  please  comment.) 

e) 

Dizziness?  (If  yes,  please  comment.) 

f) 

Disorientation?  (If  yes,  please  comment.) 

g) 

Fatigue?  (If  yes,  please  comment.) 

h) 

Glare?  (If  yes,  please  comment.) 

i) 

Other?  (If  yes,  please  comment .) 

j) 

Did  you  identify  any  potential  safety  or  health  hazards  that  may  have  posed  a 
threat  to  you  or  anyone  else  around  you?  (If  yes,  please  comment.) 

Comments: 
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Survivability,  Safety,  and  Health  Hazards 


Rate  the  system  design  for  Survivability,  Safety,  and  Health  Hazard  characteristics  using  the 
scale 

below. 


(place  an  X  in  the  appropriate  box) 

Totally 

Agree 

Agree 

Neither 
Agree  nor 
Disagree 

Disagree 

Totally 

Disagree 

NA 

a)  Light  emissions  pose  no 

survivability  risk  to  me  on  the 
battlefield. 

b)  Noise  emissions  pose  no 

survivability  risk  to  me  on  the 
battlefield. 

c)  Overall,  the  system  has  no 
adverse  impact  on  my 
survivability  in  combat. 

d)  The  system  is  safe  to  operate. 

e)  The  system  does  not  have  any 
health  hazards  associated  with  it. 

Comments: 


Rate  the  overall  system  using  the  scale  below. 


(place  an  X  in  the  appropriate  box) 

Totally 

Agree 

Agree 

Neither 
Agree  nor 
Disagree 

Disagree 

Totally 

Disagree 

NA 

a)  Overall,  the  system  enhances  my 
ability  to  complete  my  mission. 

b)  Overall,  the  system  is  well  designed 
and  is  easy  to  operate. 

c)  The  system  is  reliable. 

d)  I  would  not  hesitate  to  take  the  system 
into  combat  as  it  is  currently 
configured. 

Comments: 
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Intentionally  left  blank. 
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Appendix  B.  National  Aeronautics  and  Space  Administration-Task  Load  Index 

(NASA-TLX)  Workload  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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NASA-TLX  Workload  Questionnaire 


For  each  pair,  select  the  one  element  that  is 
more  important  for  measurement  of  workload 

for  the  task. 

Mental  Demand 

/ 

Physical  Demand 

Mental  Demand 

/ 

Temporal  Demand 

Physical  Demand 

/ 

Temporal  Demand 

NASA-TLX  RATING  SCALE  DEFINITIONS 


MENTAL  DEMAND  Low/High  How  much  mental  and  perceptual  activity 

was  required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving? 

PHYSICAL  DEMAND  Low/High  How  much  physical  activity  was  required 

(e.g..  pushing,  pulling,  turning,  controlling,  activating,,  etc.)? 
Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or 
strenuous  restful  or  laborious? 


TEMPORAL  DEMAND  Low /High  How  much  time  pressure  did  you  feel  due  to 

the  rate  or  pace  at  which  the  tasks  or  task  elements  occurred? 
Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 


NASA-TLX  Workload  Questionnaire 


For  each  workload  element  listed  below,  please  indicate  (with  an  exact  mark  on  the  line)  how  much 
each  element  contributed  to  your  overall  workload  experienced  in  the  task  you  just  performed.  Please 
write  the  corresponding  number  for  your  mark  in  the  space  provided  below  each  line. 

Mental  Workload  I - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 

0  100 

Low  High 

What  number  is  this? _ 
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Physical  demand 


What  number  is  this? 


Temporal  Demand 


100 

High 


What  number  is  this? 
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Intentionally  left  blank. 
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Appendix  C.  Modified  Cooper-Harper 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Appendix  D.  Situation  Awareness  Rating  Technique 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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SA1.  Situation  Awareness  is  defined  (simplistically)  as  “timely  knowledge  of  what  is  happening 
as  you  perform  your  tasks  during  the  mission.” 


Situation  Awareness  Rating  Technique  (SART) 

DEMAND 

Instability  of  Situation 

Likeliness  of  situation  to  change  suddenly 

Variability  of  Situation 

Number  of  variables  which  require  your  attention 

Complexity  of  Situation 

Degree  of  complication  (number  of  closely  connected  parts)  of  the  situation 

SUPPLY 

Arousal 

Degree  to  which  you  are  ready  for  activity 

Spare  Mental  Capacity 

Amount  of  mental  ability  available  to  apply  to  new  tasks 

Concentration 

Degree  to  which  your  thoughts  are  brought  to  bear  on  the  situation 

Division  of  Attention 

Amount  of  division  of  your  attention  in  the  situation 

UNDERSTANDING 

Information  Quantity 

Amount  of  knowledge  received  and  understood 

Information  Quality 

Degree  of  goodness  or  value  of  knowledge  communicated 

Familiarity 

Degree  of  acquaintance  with  the  situation 

Rate  the  level  of  each  component  of  situation  awareness  that  you  had  when  you  performed 
‘offensive  position’  tasks  as  part  of  your  unit  that  you  just  completed.  Circle  the  appropriate 
number  for  each  component  of  situation  awareness  (e.g.,  complexity  of  situation)  using  the 
aforementioned  definitions. 


Instability  of  situation: 
Variability  of  situation: 
Complexity  of  situation: 


DEMAND 

Low  1 - 2 - 3 - 4— . -5 - 6 - 7  High 

Low  1 . —2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4- . -5 . — 6— . -7  High 
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SUPPLY 


Arousal: 

Spare  mental  capacity: 
Concentration: 
Division  of  attention: 


Information  quantity: 
Information  quality: 
Familiarity: 


Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 . —4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 


UNDERSTANDING 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1— . -2 - 3 - 4- . —5 . -6- . -7  High 
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Intentionally  left  blank. 
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Appendix  E.  Sample  Situation  Awareness  Questions 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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1.  What  is  the  status  of  the  platoon  sensors? 

2.  On  the  map  plot  the  threat  mortar  location. 

3.  What  is  the  most  significant  threat  to  your  force  at  this  time? 

4.  What  action  will  the  enemy  take  in  the  next  15  minutes? 


30 


NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMAL  HRA 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 

320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

2  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI 

T  DAVIS 

J  HANSBERGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  DG  K  GUNN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 

AWC  FIELD  ELEMENT 
RDRL  HRM  DI  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 


1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  CU 
6501  E  11  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 

HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  E  REDDEN 
6450  WAY  ST 
BLDG  2839  RM  310 
FORT  BENNING  GA  31905-5400 
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NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

3  DIR  USARL 
RDRL  HR 
L  ALLENDER 
RDRL  HRM 

P  S  A V AGE-KNEPS  HIELD 
RDRL  HRS  D 
B  AMREIN 
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